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Background: The non-response rates in surveys are increasing which is problematic as it means that a progressively 
smaller proportion of the population represents the majority, and it is uncertain how health survey results are 
affected. This follow-up was performed on the non-responders to the postal questionnaire in the public health 
survey Life and Health, conducted in Orebro County Council, Sweden, where large differences in response rates 
had been found between different socio-demographic groups and geographical areas. The main objective was to 
analyse non-response bias regarding self-rated health. Methods: This follow-up study was conducted as a census to 
all non-responders in the area that had the lowest response rate and, in one other geographical area used as a 
control. It was carried out by telephone interviews, 49.3% (580 individuals) answered the follow-up. The outcome 
variable was self-rated health, a main variable in public health surveys. Differences in response patterns between 
responders and initial non-responders were approximated by prevalences with confidence intervals and adjusted 
odds ratios. Results: Poor health was more common in the initial non-response group than among the responders, 
even with consideration given to sex, age, country of birth and education. However, good health was equally 
common among responders and initial non-responders. Conclusions: Public health surveys can be biased due to 
certain groups being under-represented or not represented at all. For this reason, in repeated public health 
surveys, we recommend selective follow-ups of such groups at regular intervals. 



Introduction 

In order to be able to say anything about public health and its 
determinants, national and regional surveys are carried out and 
these are then used as the basis for planning and research. For a 
number of years, however, the number of people who participate in 
these surveys has diminished which is both a national and interna- 
tional trend in developed countries. 1 This is problematic as it means 
that a progressively smaller proportion of the population represents 
the majority, and it is uncertain how survey results are affected. In 
the Swedish national public health survey 'Health on Equal Terms', 
which is a postal questionnaire, the non-response level increased 
from 39% in 2004 to 48% in 2009. 2 Another postal public health 
survey 'Life and Health' carried out since 2000 in five Swedish 
counties had a non-response level of 35% in that year. 3 In 2004, 
it had increased to 36% 4 and to 41% in 2008. 5 In Sweden, 
the non-response in The Labour Force Surveys has increased from 
<2% in the year 1970 to >18% in 2008, which means a non-response 



increase of ~0.44 percentage points annually. 6 Internationally, 
Morton et al. have calculated that in cross-sectional surveys, there 
has been an average reduction of 0.67 percentage points per year in 
response rate during the period 1970-2003. 7 

The non-responders which are the focus here consist of people 
who for various reasons do not participate in the survey. The 
problem is that one does not know if those who do respond are rep- 
resentative of the groups being investigated as the non-responders 
can be deviating in one or several respects. The main reasons for 
non-response are that the selected people refuse to participate or 
that they are not reached. These people represent two different 
groups whose non-participation can have different consequences 
for the survey results. 8 ' 9 

We know from the Life and Health surveys that women respond 
to a higher degree than men, the middle-aged more than the young, 
individuals with a higher level of education more than those with a 
lower level, Swedish-born more than foreign-born and those living 
in residential areas more than those in blocks of flats. 10 This 
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therefore entails a bias in the responses regarding demographic and 
socio-economic conditions. This has been confirmed in a number of 
other studies. 11-1 5 There are also studies which show that 
non-responders have higher levels of morbidity and poor 
self-rated health, 16 higher levels of mortality 17 and higher propor- 
tions of daily smokers. 18 

Several studies show that one can not assume that higher levels of 
response provide different and 'safer' results, and concentrating on 
the response rate is therefore insufficient to determine the quality of 
survey data. 19 In a meta-analysis of 59 surveys, no clear relation was 
found between the level of response rate and non-response bias. 9 ' 20 
Certain studies with a response rate ~20% showed a similar bias to 
studies with >70% response rate. Deviations were more likely related 
to the variables being investigated. In line with this, Stang and 
Jockel 21 confirm that studies with a low response rate can have 
less systematic bias than those with high response rate, especially 
in situations in which the prevalence of a phenomenon increases 
after various collection waves in a study. Therefore, there exists no 
minimum level of response rate at which the non-response bias is 
expected to be too large to make inferences about a population from 
a survey sample. And correspondingly, it is not possible to say that 
there is an upper response rate at which the results are not unevenly 
distributed due to non-response. 

What is known from a number of studies is that prevalence 
estimates can be influenced by non-response but associations are 
not necessarily affected. For example, it is confirmed in the Oslo 
Health Study that unhealthy people did not participate to the same 
degree as others but that the association between social inequalities 
in health in relation to various socio-demographic conditions 
remained. 22 A Finnish study shows that survey non-response did 
not seriously bias analysis of social class inequalities in sickness 
absence and possibly health inequalities more generally. 23 In de 
Winters' study, response rate increased from 66% to 76% which 
changed the prevalence of mental health but this had no significance 
in the association found between individual characteristics and 
mental health. 24 

Health surveys are usually carried out using probability sampling 
in which there is a known degree of probability of individuals being 
included. The statistical methods widely used for population 
estimates presume that the entire sample will participate in the 
survey. As we know, this is not the case and instead, we must deal 
with a large degree of non-response that may generate bias. Potential 
non-response bias can, to some extent, be estimated and/or 
corrected for in various ways. An approach relies on standard stat- 
istical methods for treatment of non-response in surveys: imputation 
or calibration weighting. Other approaches are either to estimate the 
non-response bias by comparing early and late responders or to 
conduct a non-responders follow-up. When follow-ups are carried 
out, they usually take the form of a new probability sample. An 
alternative method which we have used in this study is to carry 
out a selective follow-up which is performed as a census of the 
non-responders in the area with the lowest response rate. 

A key objective of this study is to investigate whether there is any 
non-response bias regarding self-rated health. Further objectives are 
to study the generalizability of the results for areas with a different 
socio-demographic structure, to identify effects of calibration 



weightings, and to study the change of response rates for different 
socio-demographic groups by using a follow-up. 

Methods 

The survey Life and Health is a public health survey which has been 
carried out by five county councils in Sweden in the years 2000, 2004 
and 2008. It addresses persons in the age group of 18-84 years with 
questions on health, lifestyle and living conditions. The sampling 
procedure used was a random sample stratified by sex, age group 
and geographical area. The data were collected by a postal question- 
naire. In 2004, Orebro County Council decided to carry out a 
non-response follow-up as large response rate variations were 
evident between geographical areas and different socio-demographic 
groups. 

In this study, it was decided to make a selective sample without 
the use of probability sampling in order to study the area 
characterized by especially low response frequencies as a whole, 
but also within various socio-demographic groups. The area 
concerned consists of blocks of flats, has many people on low 
incomes, many with a low level of education, foreign-born and 
single-parent households and in this respect it is an unfavourable 
area in Orebro municipality. There is evidence that there is an as- 
sociation between low response rate and deprived areas. 15 ' 22 The 
'Study area' consisted of non-responders who lived in blocks of 
rental flats in the municipality of Orebro, Sweden. This grouping 
of neighbourhoods has been used in previous studies 10 ' 25 ' 26 and 
these areas consist of small administrative geographical areas, 
key-codes, with similar characteristics but not necessarily adjacent. 
Another of the county's medium-sized municipalities was selected in 
order to function as a 'Control area'. The sample size in these two 
areas was 3120 persons (population size 28 484). After identifying 
over coverage in the initial sample of 104 persons, non-responders in 
these areas amounted to 1188 and all were included in the follow-up. 
The study population is described in more detail in table 1. 

The telephone interview 

During November-December 2004, Statistics Sweden carried out a 
follow-up among everyone in the selected areas who had not 
responded to the postal questionnaire. Those contacted were 
requested in a telephone interview (after up to five contact 
attempts by telephone), to answer a limited number of questions 
from the questionnaire. 

The individuals in the sample were informed that the information 
regarding sex, age, geographical area, education level, occupation 
and native country obtained from the Swedish official registries 
would be linked to their answers if they responded to the 
interview. The responders thus accepted the linking of official 
registry data to the questionnaire data by informed consent. The 
personal identification numbers were deleted directly after the 
record linkage with the national registers and the interview data 
are thus anonymous. 

After identifying an additional over-coverage of 12 persons the 
non-responders in the follow-up were 596 persons. There were 326 
persons who for various reasons could not be reached, 96 persons 



Table 1 Population and sample size, response rate, number of initial responders and initial non-responders in different areas 



Population Survey sample Initial responders Initial non-responders 

N n (response rate, %) m r "W, 



Region 1043350 68460 (64) 43 589 

County 208091 17160 (63) 10781 

Study area 20777 2080 (58) 1207 359 

Control area 7707 1040 (60) 621 221 



Population (W). survey sample size (n), initial response rate (after consideration of over-coverage), initial responders (postal survey) (m r ) and 
initial non-responders (interview) (m n _ r ) for study and control area. 
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who were unable to participate due to disability or difficulties with 
the language and 172 persons who refused to participate in the 
survey, 2 persons claimed that they had participated in the main 
study. A total of 580 of the initial non-responders answered to the 
interview, response rate 49. 3%. 27 

Measures 

In this study, self-rated health is used as the outcome variable. This 
was measured, both in the postal questionnaire and the interview, 
with the question 'How do you rate your general health?' with the 
options 'Very good', 'Good', 'Neither good nor poor', 'Poor' and 
'Very poor'. The two last alternatives were used to estimate poor 
self-rated health and the first two to estimate good self-rated health. 
Other data used were registry data from Statistics Sweden regarding 
the responders' sex, age, native country, education and employment 
status. In order to avoid too small study groups, country of birth is 
divided into three groups: people born in Sweden, people born 
outside Sweden but in the Nordic countries (Denmark, Finland, 
Iceland, Norway) and people born outside the Nordic countries. 
Education is categorized into low level of education (<9 years), 
medium level (comprehensive school up to 12 years), high level 
(>12 years) and unknown. The last category applies mainly to 
some 75- to 84-year-olds and people born outside the Nordic 
area. Employment status is divided into four categories: gainfully 
employed, student, unemployed and other. The last group 
includes disability pensioners, retired people, those on parental 
leave and domestic workers. 

Statistical procedures 

Population weightings have been created in order to estimate preva- 
lences at the population level. For questionnaire data, calibration 
weightings have been calculated. 28 ' 29 The weightings were 
produced by Statistics Sweden with the help of information from 
registers of the total population. Apart from adjustment for various 
selection sizes in the different strata, the register data are used for 



calibration of non-response bias for various groups of individuals. 
The calibration weightings are constructed by use of population level 
information on a set of auxiliary variables to adjust the design 
weightings. The calibrated weightings are based on auxiliary 
population totals and the individual value for the auxiliary 
variables for the survey responders. The register data used to 
produce the calibration weightings are sex, age, education level, 
employment status, country of birth and year of immigration. 

The differences in the response patterns for self-rated health 
between the responders vis-d-vis the initial non-responders and 
the influence of calibration weightings are approximated by 
studying the overlap of the point estimates [95% confidence 
intervals (95% CI), SAS 9.2, Proc survey means]. Logistic 
regression (SPSS 17.0, Enter method) is used to study the effect of 
the initial non-responders compared with the responders adjusted 
for sex, age, country of birth and education with the dependent 
variable self-rated health. By using these methods in Study and 
Control area, the differences in response patterns are used to 
analyse the generalizability of the results for areas with different 
socio-demographic structure. The response rates for different 
socio-demographic groups before and after the follow-up are 
presented by percentage points. 

Results 

The results indicate that poor health is significantly more prevalent 
in the initial non-response group compared with responders in the 
total group for both the Study and Control areas (table 2). This is 
also the case among women in the Study area. When the responses 
from the initial non-responders are added to the responders, there is 
no longer any indication that the proportions of poor health differ 
between the two groups. The results also show that there are certain 
differences between the areas studied regarding level of education 
and employment status (not in table). Among the initial 
non-responders in the Study area, the proportion with low educa- 
tional level is significantly higher (95% CI for initial non-responders: 
29.2-40.4 and responders: 21.7-28.6) and the proportion of 



Table 2 Percentage proportion (P) and 95% CI for the proportion to have poor/very poor self-rated health or good/very good self-rated 
health by sex, response group and area 



Study area Control area 



P (%) 95% CI P (%) 95% CI 



Poor/very poor self-rated health 

Women Responders 

Initial non-responders 
Responders+initial non-responders 
Responders calibrated 

Men Responders 

Initial non-responders 
Responders+initial non-responders 
Responders calibrated 

Total Responders 

Initial non-responders 
Responders+initial non-responders 
Responders calibrated 
Good/very good self-rated health 



Women 



Men 



Total 



Responders 

Initial non-responders 

Responders+initial non-responders 

Responders calibrated 

Responders 

Initial non-responders 

Responders+initial non-responders 

Responders calibrated 

Responders 

Initial non-responders 

Responders+initial non-responders 

Responders calibrated 



9.5 
22.4 
12.0 
10.3 
12.0 
17.0 
13.4 
12.9 
10.7 
19.8 
12.7 
11.6 

65.7 
55.8 
63.6 
64.7 
70.3 
67.1 
69.2 
69.4 
67.9 
61.3 
66.3 
67.0 



7.05-11.87 
15.87-28.91 
9.61-14.34 
7.66-12.99 
8.65-15.31 

11.62- 22.41 

10.63- 16.26 
9.18-16.58 
8.63-12.70 

15.54-24.07 
10.86-14.51 
9.30-13.80 

61.66-69.75 
47.84-63.74 
59.98-67.17 
60.57-68.83 
65.63-75.06 
60.80-73.48 
65.50-72.83 
64.48-74.30 
64.84-71.01 
56.12-66.38 
63.70-68.84 
63.76-70.14 



7.7 
17.8 
9.7 
8.8 
4.9 
12.7 
7.0 
5.4 
6.3 
15.1 
8.3 
7.0 

65.7 
62.4 
65.1 
63.7 
72.0 
71.7 
72.4 
71.1 
68.9 
67.3 
68.9 
67.5 



4.73-10.72 
10.28-25.24 
6.80-12.64 
5.38-12.19 

2.43- 7.34 
5.92-19.52 

4.44- 9.50 
2.64-8.08 
4.34-8.19 

10.09-20.16 
6.37-10.22 
4.85-9.18 

60.46-70.92 
53.08-71.79 
60.57-69.68 
58.31-69.12 

66.78- 77.16 
63.08-80.26 
68.16-76.70 
65.75-76.42 
65.24-72.61 
60.93-73.60 

65.79- 72.03 
63.72-71.33 
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Table 3 OR and 95% CI for for having poor/very poor or good/very 
good self-rated health adjusted for sex, age, country of birth, 
educational level and employment status by area 



Poor/very poor self-rated health 


Study area 


Control area 




OR (95% CI)* 


OR (95% CI)** 


Responders 


Ref. 


Ref. 


Initial non-responders 


2.01 (1.43-2.82) 


2.33 (1.35-4.03) 


Women 


Ref 


Ref 


Men 


0.86 (0.63-1.16) 


0.70 (0.41-1.18) 


Age (years) 






18-34 


Ref. 


Ref. 


35-49 


3.40 (1.96-5.89) 


1 .36 (0.50-3.70) 


50-64 


3.77 (2.16-6.59) 


4.01 (1.70-9.47) 


65-84 


3.12 (1.72-5.69) 


1.16 (0.42-3.22) 


Sweden 


Ref 


Ref 


Nordic countries 


1.80 (0.89-3.65) 


2.05 (0.81-5.17) 


Outside Nordic countries 


2.30 (1.62-3.28) 


2.21 (0.78-6.26) 


Higher educational level 


Ref. 


Ref. 


Medium educational level 


1.33 (0.80-2.20) 


1.10 (0.42-2.88) 


Low educational level 


2.28 (1.36-3.83) 


2.08 (0.78-5.53) 


Educational level unknown 


2.49 (1.32-4.71) 


6.73 (2.00-22.64) 


Good/very good self-rated health 


OR (95% CI)*** 


OR (95% CI)**** 


Responders 


Ref. 


Ref. 


Initial non-responders 


0.84 (0.65-1.10) 


1.03 (0.72-1.48) 


Women 


Ref 


Ref 


Men 


1.20 (0.97-1.49) 


1.38 (1.01-1.88) 


Age (years) 






18-34 


Ref. 


Ref. 


35-49 


0.39 (0.27-0.54) 


0.59 (0.35-0.99) 


50-64 


0.33 (0.23-0.46) 


0.29 (0.18-0.48) 


65-84 


0.28 (0.19-0.40) 


0.26 (0.15-0.44) 


Sweden 


Ref. 


Ref. 


Nordic countries 


0.47 (0.26-0.84) 


1.02 (0.54-1.91) 


Outside Nordic countries 


0.78 (0.59-1.02) 


0.40 (0.20-0.83) 


Higher educational level 


Ref. 


Ref. 


Medium educational level 


0.71 (0.52-0.98) 


0.62 (0.37-1.05) 


Low educational level 


0.54 (0.38-0.76) 


0.47 (0.27-0.82) 


Educational level unknown 


0.46 (0.30-0.71) 


0.36 (0.18-0.72) 



*Multiple Logistic regression, n = 1524, Nagelkerke R 2 0.114; 
**Multiple Logistic regression, n = 820 Nagelkerke R 2 0.125; 
***Multiple Logistic regression, n = 1524, Nagelkerke R 2 0.113; 
****Multiple Logistic regression, n = 820, Nagelkerke R 2 0.129 
The Odds ratios that are significantly above or under one are all 
marked in bold. 

gainfully employed significantly lower (95% CI for initial 
non-responders: 36.6-47.0 and responders: 47.8-55.1). This is not 
the case in the Control area. The calibration weighting of the 
responders tend to increase the proportion of poor health and 
decrease the proportion of good health. 

The difference which exist between the 'Responders+initial 
non-responders' and the 'Responders' remains when compared 
with 'Responders calibrated' but the differences tend, however, to 
be somewhat smaller. 

Also in the logistic regression, adjusted for the background 
variables sex, age, country of birth and education level, it was clear 
that for those in the initial non-response group there was a higher 
probability of poor health. This was evident in both Study and 
Control area. When it comes to good health, there are no statistically 
significant variations between the response group and the initial 
non-response group (table 3). 

When consideration is given to the initial non-response group, 
the proportion of responders born in Sweden, as well as those born 
outside the Nordic countries increased by more than 20 percentage 
points. The highly educated had the highest response rate in the 
postal questionnaire and this also is the case when complemented 
with the responses from the non-response interviews. For those with 
a low level of education, however, the response rate increased by a 
higher degree. Men increased their response rate more than women 



Table 4 Response rates (percent) by sex, age, education level, 
employment status and country of birth for Responders and Total 
response (Responders + initial non-responders responders) in Study 
and Control area 





Dficnnnnorc 


Total 








Women 


62 


80 


Men 


49 


74 


Age (years) 






18-34 


48 


76 


35^19 


53 


74 


50-64 


65 


80 


65-79 


65 


79 


80-84 


59 


72 


Low educational level 


48 


72 


Medium educational level 


58 


80 


Higher educational level 


67 


86 


Educational level unknown 


48 


60 


Gainfully employed 


61 


82 


Student 


45 


76 


Unemployed 


59 


82 


Other employment status 


51 


68 


Born in Sweden 


59 


80 


Born outside Sweden but in the Nordic countries 


57 


73 


Born outside the Nordic countries 


44 


68 



and the younger more than the older (table 4). Especially the group 
younger men who were the most under-represented in the question- 
naire survey achieved a response rate of 70%, which represents an 
increase of 33 percentage points (not in table). 

Discussion 

The main reason this follow-up was performed on parts of the 
non-response group in the 'Life and Health' survey was the large 
differences in response rates found between different 
response-groups and areas, and also the general decline in 
response rates in public health surveys experienced both nationally 
and internationally in developed countries. 1 ' 2 ' 7 This situation is 
problematic as it means that a progressively smaller proportion of 
the population represents the majority, and it is uncertain how this 
affects survey results. It is questionable if it is possible to rely on the 
results of public health surveys that have a non-response rate as large 
as >40%, which is the case in the 'Life and Health' survey. We 
decided to carry out this follow-up study as a census in the area 
that had the lowest total response rate as a whole and also in relation 
to socio-demographic groups where we expected to find the greatest 
variations in the response pattern between responders and 
non-responders. 

In the study that was carried out by telephone interviews, 49.3% 
of the follow-up sample answered. In comparison with several other 
non-response follow-ups, this is a high proportion. In various 
Swedish non-response surveys carried out during the 21st century 
concerning public health questionnaires, the proportion has varied 
between 7% and 27%. 2 The comparatively high number of contact 
attempts (five) and the small number of questions asked were 
certainly factors that contributed to increasing the response rate. 
When the initial non-responders were added to the responders, 
the total number of the response group was substantially 
increased. In spite of this there are still half of the non-responders 
we know nothing about which is a weakness. There is a risk that 
these people deviate both in regard to the socio-demographic data 
and to the response pattern from those who responded. 30 The most 
common reason for the selected persons not being included in the 
follow-up was that they were not reached (54.7%), which to a large 
extent was due to the fact that there was a lack of information 
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regarding telephone numbers. This is a factor that can be expected to 
be of increasing importance as the number of people with a fixed 
telephone is diminishing in the population. 

Self-rated health is an important variable in public health surveys 
and poor health is often used as the outcome measure in analyses of 
associations with different determinants of health. The results show 
an underestimation of poor health in the initial non-response group 
which is in line with other surveys. 16 ' 22 This is the case even with 
consideration given to sex, age, country of birth and level of 
education. It does not, however, apply to good health where there 
was no significant difference between questionnaire responders and 
the initial non-response group either for the proportion with good 
health or for the adjusted odds ratio (OR). Depending on which part 
of the variable that is studied, the result is therefore different. This is 
interesting considering that the salutogenic perspective is the main 
focus in public health promotion. Here, the starting point is the 
determinant for good health and not poor health. 

In the comparison of the two areas, significant differences are seen 
between those who responded to the main survey postal question- 
naire and those who responded in the interviews of initial non-re- 
sponders concerning level of education and employment status. The 
initial non-responders in the Study area had lower level of 
education, lower proportion of gainfully employed and a higher 
proportion of unemployed. 31 This can but does not necessarily 
lead to systematic errors in the results. 9 ' 20 The differences found in 
the Study area could not be seen in the Control area. A conclusion 
that can be reached from this is that one should be careful 
concerning generalizations regarding other areas with different 
population structures. 

In the 'Life and Health' survey, calibration weighting was used to 
adjust for the non-response. 28 ' 29 The statistically significant 
variations which exist between the initial non-responders and the 
questionnaire responders remained even when compared with the 
calibration-weighted questionnaire responses but the differences 
tend, however, to be somewhat smaller. If calibration weightings 
are provided using good auxiliary variables, this is a more 
cost-effective method than adjusting for these differences by using 
follow-ups. It is a question of resource allocation in large-scale 
surveys. It is also plausible that the two different methods, postal 
questionnaire and telephone interview, themselves generate different 
responses when it comes to questions about health. 32-34 From the 
results, it is clear that those in the initial non-response group had a 
higher probability of poor health. This is in line with other 
studies. 32 ' 35 The use of calibration weighting instead of comple- 
menting the questionnaire data with interview data from the 
non-response survey means that one avoids the uncertainty that 
results from the combination of the two methods. Due to the 
reasons mentioned, we therefore recommend calibration 
weighting for the adjustment of variations in response frequencies 
in large-scale surveys. The use of non-response follow-ups in 
interview form should mainly be used to estimate non-response 
bias, to increase understanding and to facilitate the interpretation 
of the results obtained. 

The follow-up contributed to an increased response rate in all 
socio-demographic groups. It also decreased the large differences 
that were seen from the postal questionnaire between men and 
women, younger and older, lower and higher levels of education, 
Swedish-born and foreign-born and between various employment 
groups. Variations in the likelihood of response between different 
groups can accordingly be levelled out with follow-ups in which a 
small number of questions are put in telephone interviews. 

Can we use the results from public health surveys that have pro- 
gressively lower response frequencies? Yes, we believe we can. We 
must be aware, however, that there can be bias in the results due to 
certain groups being under-represented or perhaps not represented 
at all. This is especially important when studying certain character- 
istics for which there can be assumed to exist bias, such as poor 



health. For this reason in public health surveys, we recommend 
selective follow-ups of under-represented groups at regular intervals. 
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